Implementing clarification dialogues in open domain question answering

We examine the implementation of clarification dialogues, a mechanism for ensuring that question answering systems take into account user goals by allowing them to ask series of related questions either by refining or expanding on previous questions with follow-up questions, in the context of open domain Question Answering systems. We develop an algorithm for clarification dialogue recognition through the analysis of collected data on clarification dialogues and examine the importance of clarification dialogue recognition for question answering. The algorithm is evaluated and shown to successfully recognize the start and continuation of clarification dialogues in 94% of cases. We then show the usefulness of the algorithm by demonstrating how the recognition of clarification dialogues can simplify the task of answer retrieval. 1 Clarification dialogues in Question Answering Open domain Question Answering (QA) Systems, as defined for example in the TREC evaluation framework which currently provides a widely accepted standard baseline for research in this area (see Voorhees 2002 for an outline of the framework and an overview of current systems), aim to determine an answer to a question (which is not limited to a particular task or topic) by searching for a response in a collection of documents such as newspaper articles. In order to achieve this (see for example Harabagiu, Moldovan, Pasca, Surdeanu, Mihalcea, Gîrju, Rus, Lacatusu, Morarescu and Bunescu 2002), systems narrow down the search by using information retrieval techniques to select a subset of documents, or paragraphs within documents, containing keywords from the question and a concept which corresponds to the correct question type (e.g. a question starting with the word “Who?” would require an answer containing a person). The exact answer sentence is then sought by either attempting to unify the answer semantically with the question, through some kind of logical transformation (e.g. Moldovan and Rus 2001) or by some form of pattern matching (e.g. Soubbotin 2002; Harabagiu et al. 1999). Often, though, a single question is not enough to meet users’ goals: a wider dialogue (which, in the case of TREC-style QA systems is limited to a series of question/answer pairs, and not, as happens in human dialogue, also question/question pairs), either elaborating and building on information gathered, or clarifying previously given information is required, i.e. a dialogue which will enable the users to fully achieve their informational goals. We shall hence refer to such exchanges as clarification dialogues, following the terminology used for example by Purver et al. (2003; 2002) and Ginzburg (1998), as the questions that constitute them either clarify previous questions or answers or clarify the mental picture the user is trying to build by elaborating on previously asked or given information: the expression “clarification dialogue” indicates that we are in fact a) examining a dialogue, albeit a very limited one, where only one party in the dialogue asks questions and only one party gives answers; and b) we are considering a dialogue which clarifies some concept in the questioner’s mind, whether this be by asking for some new information related to the topic investigated or asking for an explanation of something already given. One example of a clarification dialogue is in the form of questions which seek to clarify the meaning of an answer, for example when the user has not understood a term contained in the answer, as in the following exchange: (1) Q1: What is a fairy tale? A1: The American Heritage dictionary tells me it is a fanciful tale of legendary deeds and creatures. Q2: What does fanciful mean? A2: ... On other occasions users want to expand on a given answer in order to have more details, as in the following example, where the user, having discovered a need (the necessity to have a license to fish) wants more details about how to go about fulfilling that need (the cost of the license): (2) Q1: Do I need a license to fish in the Tiber river? A1: Yes. Q2: How much? A2: ... In other cases the user’s goal is to form a broad picture about some topic and a number of separate questions are needed in order to achieve the breadth of information required: (3) Q1: Where was Frank Sinatra born? A1: Hoboken, N.J. Q2: Where did he grow up? A2: Hoboken, N.J. Q3: What kind of childhood did he have? A3: ... The common link between the above dialogue fragments is the fact that the question/answer sequences form coherent units of discourse quite different from an interaction such as the following: (4) Q: What is caffeine? A: A stimulant. Q: What imaginary line is halfway between the North and South Poles? A: The equator. Q: Where is John Wayne airport? A: ... In (4) there is no relationship between the questions or between the questions and previous answers and hence in seeking an answer there is no immediate reason to take into consideration previously asked questions or previously given answers. In fragments (1) to (3), however, in order to answer the questions correctly it is necessary to take into consideration the previous context in order to satisfy the user’s goals. In (1), for example, the user isn’t asking for the generic meaning of the word fanciful (the American Heritage Dictionary, for example, gives three separate meanings for the word fanciful) but the specific meaning that word takes in the sentence “it is a fanciful tale of legendary deeds and creatures”. Similarly in (2) the question “How much?” makes no sense, and cannot be answered, without reference to the context. Exchanges such as those in examples (1) to (3) therefore have in common the feature that to answer a question satisfactorily some reference must be made to previously asked questions and previously given answers. While a number of researchers have looked at clarification dialogue from a theoretical point of view (e.g. Purver et al. 2003; Purver et al. 2002; Ginzburg 1998; Ginzburg and Sag 2000; van Beek at al. 1993), or from the point of view of task oriented dialogue within a narrow domain (e.g. Ardissono and Sestero 1996), there has been little work on clarification dialogue for open domain question answering systems such as the ones presented at the TREC workshops, where the task that the user is pursuing and the subject matter of the user’s investigations are not known a priori. Initial work in this direction has consisted of a series of experiments carried out for the (subsequently abandoned) “context” task in the TREC10 QA workshop (Voorhees 2002; Harabagiu et al. 2002) and the initial experiments presented by De Boni and Manandhar (2003b). Here we seek to partially address this problem by looking at a particular aspect of clarification dialogues in the context of open domain question answering: the problem of recognizing that a clarification dialogue is occurring, i.e. how to decide whether the current question is part of an on-going series (i.e. clarifying previous questions) or the start of a new series; we then show how the recognition that a clarification dialogue is occurring can simplify the problem of answer retrieval.